Search for: All records

Ethical concerns around AI have increased emphasis on model auditing and reporting requirements. We thoroughly review the current state of governance and evaluation prac- tices to identify specific challenges to responsible AI devel- opment in OSS. We then analyze OSS projects to understand if model evaluation is associated with safety assessments, through documentation of limitations, biases, and other risks. Our analysis of 7902 Hugging Face projects found that while risk documentation is strongly associated with evaluation practices, high performers from the platform’s largest com- petitive leaderboard (N=789) were less accountable. Recog- nizing these delicate tensions from performance incentives may guide providers in revisiting the objectives of evaluation and legal scholars in formulating platform interventions and policies that balance innovation and responsibility. 

Abstract The Institutional Grammar (IG) is a rigorous tool for analyzing the laws and policies governing nonprofit organizations; however, its use was limited due to the time-consuming nature of hand-coding. We introduce an advance in Natural Language Processing using a semantic role labeling (SRL) classifier that reliably codes rules governing and guiding nonprofit organizations. This paper provides guidance for how to hand-code using the IG, preprocess text for machine learning, and demonstrates the SRL classifier for automated IG coding. We then compare the hand-coding to the SRL coding to demonstrate its accuracy. The advances in machine learning now make it feasible to utilize the IG for nonprofit research questions focused on inter-organizational collaborations, government contracts, federated nonprofit organizational compliance, and nonprofit governance, among others. An added benefit is that the IG is adaptable for different languages, thus enabling cross-national comparative research. By providing examples throughout the paper, we demonstrate how to use the IG and the SRL classifier to address research questions of interest to nonprofit scholars. 

Sustainable Open Source Software (OSS) forms much of the fabric of our digital society, especially successful and sustainable ones. But many OSS projects do not become sustainable, resulting in abandonment and even risks for the world's digital infrastructure. Prior work has looked at the reasons for this mainly from two very different perspectives. In software engineering, the focus has been on understanding success and sustainability from the socio-technical perspective: the OSS programmers' day-to-day activities and the artifacts they create. In institutional analysis, on the other hand, emphasis has been on institutional designs (e.g., policies, rules, and norms) that structure project governance. Even though each is necessary for a comprehensive understanding of OSS projects, the connection and interaction between the two approaches have been barely explored. In this paper, we make the first effort toward understanding OSS project sustainability using a dual-view analysis, by combining institutional analysis with socio-technical systems analysis. In particular, we (i) use linguistic approaches to extract institutional rules and norms from OSS contributors' communications to represent the evolution of their governance systems, and (ii) construct socio-technical networks based on longitudinal collaboration records to represent each project's organizational structure. We combined the two methods and applied them to a dataset of developer digital traces from 253 nascent OSS projects within the Apache Software Foundation (ASF) incubator. We find that the socio-technical and institutional features relate to each other, and provide complimentary views into the progress of the ASF's OSS projects. Refining these combined analyses can help provide a more precise understanding of the synchronization between the evolution of institutional governance and organizational structure.